Subject designating device and subject tracking apparatus

ABSTRACT

A subject designating device includes: a representative value calculation unit that calculates a representative value for each image of a brightness image and chrominance images based upon pixel values indicated at pixels present within a first subject area; a second image generation unit that creates a differential image by subtracting the representative value from pixel values indicated at pixels present within a second subject area; a binarizing unit that binarizes the differential image; a synthesizing unit that creates a synthetic image by combining binary images in correspondence to the brightness image and the chrominance images; a mask extraction unit that extracts a mask constituted with a white pixel cluster from the synthetic image; an evaluation value calculation unit that calculates an evaluation value indicating a likelihood of the mask representing the subject; and a subject designating unit that designates the subject in the target image based upon the evaluation value.

INCORPORATION BY REFERENCE

The disclosure of the following priority application is hereinincorporated by reference: Japanese Patent Application No. 2011-34823filed Feb. 21, 2011.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a subject designating device and asubject tracking apparatus.

2. Description of Related Art

An imaging device in the related art determines a subject position basedupon an AF area selected by the user and executes focus adjustmentprocessing for the subject thus designated (see Japanese Laid OpenPatent Publication No. 2004-205885).

SUMMARY OF THE INVENTION

The imaging device in the related art designates the subject positionentirely based upon the AF area selected by the user without utilizingsubject color information or brightness information for purposes ofsubject position determination. This means that a subject cannot bedesignated in certain photographic scenes.

According to the 1st aspect of the present invention, a subjectdesignating device comprises: a first image generation unit thatgenerates a brightness image expressed based upon a brightness componentof a target image and chrominance images expressed based uponchrominance components of the target image; a representative valuecalculation unit that calculates a representative value for each imageof the brightness image and the chrominance images having been generatedby the first image generation unit, based upon pixel values indicated atpixels present within a first subject area containing a subject in theeach image of the brightness image and the chrominance images; a secondimage generation unit that creates a differential image each incorrespondence to the each image of the brightness image and thechrominance images having been generated by the first image generationunit, by subtracting the representative value calculated by therepresentative value calculation unit from pixel values indicated atpixels present within a second subject area containing the subject inthe each image of the brightness image and the chrominance images; abinarizing unit that binarizes the differential image having beengenerated by the second image generation unit; a synthesizing unit thatcreates a synthetic image by combining a binary image created by thebinarizing unit in correspondence to the brightness image and binaryimages created by the binarizing unit in correspondence to thechrominance images; a mask extraction unit that extracts a maskconstituted with a white pixel cluster from the synthetic imagegenerated by the synthesizing unit; an evaluation value calculation unitthat calculates an evaluation value indicating a likelihood of the mask,having been extracted by the mask extraction unit, representing thesubject; and a subject designating unit that designates the subject inthe target image based upon the evaluation value having been calculatedby the evaluation value calculation unit.

According to the 2nd aspect of the present invention, in the subjectdesignating device according to the 1st aspect, it is preferred that thesubject designating device further comprises: a third image generationunit that weights each pixel value indicated at each pixel present inthe differential image generated by the second image generation unit bymultiplying the each pixel value by a coefficient determined incorrespondence to a distance from a third subject area containing thesubject in the differential image to the each pixel present in thedifferential image so as to generate a weighted differential image incorrespondence to the each image of the brightness image and thechrominance images. The binarizing unit binarizes the weighteddifferential image having been generated by the third image generationunit.

According to the 3rd aspect of the present invention, in the subjectdesignating device according to the 1st aspect, it is preferred that therepresentative value is any one of; an average value, a most frequentvalue and a median of the pixel values indicated at the pixels presentwithin the second subject area.

According to the 4th aspect of the present invention, in the subjectdesignating device according to the 2nd aspect, it is preferred that therepresentative value is any one of; an average value, a most frequentvalue and a median of the pixel values indicated at the pixels presentwithin the second subject area.

According to the 5th aspect of the present invention, in the subjectdesignating device according to the 1st aspect, it is preferred that thesecond subject area is any one of an area containing a coordinate pointon the target image indicated by a user and an area containing an AFpoint for focus adjustment processing.

According to the 6th aspect of the present invention, in the subjectdesignating device according to the 2nd aspect, it is preferred that thesecond subject area is any one of an area containing a coordinate pointon the target image indicated by a user and an area containing an AFpoint for focus adjustment processing.

According to the 7th aspect of the present invention, in the subjectdesignating device according to the 3rd aspect, it is preferred that thesecond subject area is any one of an area containing a coordinate pointon the target image indicated by a user and an area containing an AFpoint for focus adjustment processing.

According to the 8th aspect of the present invention, in the subjectdesignating device according to the 4th aspect, it is preferred that thesecond subject area is any one of an area containing a coordinate pointon the target image indicated by a user and an area containing an AFpoint for focus adjustment processing.

According to the 9th aspect of the present invention, a subject trackingapparatus comprises: a subject designating device according to the 1staspect; and a tracking unit that tracks the subject designated by thesubject designating unit from one frame to the next through a pluralityof frames.

According to the 10th aspect of the present invention, a subjecttracking apparatus comprises: a subject designating device according tothe 2nd aspect; and a tracking unit that tracks the subject designatedby the subject designating unit from one frame to the next through aplurality of frames.

According to the 11th aspect of the present invention, a subjecttracking apparatus comprises: a subject designating device according tothe 3rd aspect; and a tracking unit that tracks the subject designatedby the subject designating unit from one frame to the next through aplurality of frames.

According to the 12th aspect of the present invention, a subjecttracking apparatus comprises: a subject designating device according tothe 4th aspect; and a tracking unit that tracks the subject designatedby the subject designating unit from one frame to the next through aplurality of frames.

According to the 13th aspect of the present invention, a subjecttracking apparatus comprises: a subject designating device according tothe 5th aspect; and a tracking unit that tracks the subject designatedby the subject designating unit from one frame to the next through aplurality of frames.

According to the 14th aspect of the present invention, a subjecttracking apparatus comprises: a subject designating device according tothe 6th aspect; and a tracking unit that tracks the subject designatedby the subject designating unit from one frame to the next through aplurality of frames.

According to the 15th aspect of the present invention, a subjecttracking apparatus comprises: a subject designating device according tothe 7th aspect; and a tracking unit that tracks the subject designatedby the subject designating unit from one frame to the next through aplurality of frames.

According to the 16th aspect of the present invention, a subjecttracking apparatus comprises: a subject designating device according tothe 8th aspect; and a tracking unit that tracks the subject designatedby the subject designating unit from one frame to the next through aplurality of frames.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the structure of a camera achieved inan embodiment.

FIGS. 2A and 2B present a flowchart of subject tracking processing.

FIG. 3 shows how the distance weighting coefficient Dist may becalculated in a first example.

FIG. 4 shows how the distance weighting coefficient Dist may becalculated in a second example.

FIG. 5 shows how a synthetic image may be created, for example.

DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 is a block diagram showing the structure of the camera achievedin an embodiment of the present invention. A camera 100 includes anoperation member 101, a lens 102, an image sensor 103, a control device104, a memory card slot 105 and a monitor 106. The operation member 101includes various input members operated by the user, such as a powerbutton, a shutter release button, a zoom button, a cross key, a confirmbutton, a review button and a delete button.

While the lens 102 is constituted with a plurality of optical lenses,FIG. 1 simply shows a single representative lens. The image sensor 103,which may be a CCD image sensor or a CMOS image sensor, captures asubject image formed through the lens 102. The image sensor 103 outputsimage signals obtained by capturing the image to the control device 104.

The control device 104, constituted with a CPU, a memory and otherperipheral circuits, controls the camera 100. It is to be noted that thememory constituting part of the control device 104 includes an SDRAM anda flash memory. The SDRAM, which is a volatile memory, is used as a workmemory where a program executed by the CPU is opened and as a buffermemory where data are temporarily recorded by the CPU. In the flashmemory, which is a non-volatile memory, program data related to theprogram executed by the control device 104, various parameters that areread for program execution and the like are recorded.

The control device 104 generates image data in a predetermined imageformat such as the MEG format (hereafter referred to as “main imagedata”) based upon the image signals input thereto from the image sensor103. In addition, the control device 104 generates display image data,e.g., thumbnail image data, based upon the image data having beengenerated. The control device 104 creates an image file that containsthe main image data and the thumbnail image data having been generatedand is appended with header information. The image file thus created isoutput to the memory card slot 105. The embodiment is described byassuming that the main image data and the thumbnail image data are bothimage data expressed in the RGB colorimetric system.

At the memory card slot 105, in which a memory card, used as a storagemedium, is inserted, an image file output from the control device 104 isrecorded as the image file is written into the memory card. In addition,in response to an instruction issue from the control device 104, animage file stored in the memory card is read at the memory card slot105.

At the monitor 106, which is a liquid crystal monitor (back sidemonitor) installed at the rear surface of the camera 100, an imagestored in the memory card, a setting menu enabling selection of settingsfor the camera 100 and the like are displayed. In addition, as the usersets the camera 100 in a photographing mode, the control device 104outputs to the monitor 106 display image data corresponding to imagesobtained from the image sensor 103 in time series. As a result, alive-view image corresponding to the display image data is displayed atthe monitor 106.

The control device 104 in the embodiment designates an object within thephotographic image plane as a subject based upon initial subjectpositions in the photographic image plane, subject color information andsubject brightness information, and tracks the designated subject. FIGS.2A and 2B present a flowchart of the subject tracking processingexecuted in the embodiment. The processing shown in FIGS. 2A and 2B isexecuted by the control device 104 as a program started up as an imagedata input from the image sensor 103 starts in response to a subjecttracking start instruction issued by the user.

In step S10, the control device 104 reads an image input from the imagesensor 103 as a target image. At this time, the control device 104reduces the size of the image input from the image sensor 103 to, forinstance, a 360×240 (pixel) size and designates the reduced image as thetarget image. This allows the subsequent processing to be executedquickly. As explained earlier, the image data input from the imagesensor 103 are expressed in the RGB colorimetric system in theembodiment. Accordingly, RGB image data are converted to a brightnessimage expressed based upon a brightness component (Y component) andchrominance images expressed based upon chrominance components (Cbcomponent and Cr component) in a YCbCr color space, as expressed in (1)through (3) below or (4) through (6) below. It is to be noted that thisconversion processing does not need to be executed if the image datainput from the image sensor 103 are already expressed in the YCbCrcolorimetric system.Y=0.2990R+0.5870G+0.1140B  (1)Cb=−0.1687R−0.3313G+0.5000B+128  (2)Cr=0.5000R−0.4187G−0.0813B+128  (3)Y=0.25R+0.50G+0.25B  (4)Cb=−0.25R−0.50G+0.75B+128  (5)Cr=0.75R−0.50G−0.25B+128  (6)

Once the processing in step S10 is executed, the operation proceeds tostep S20, in which the control device 104 sets a limit value SmallLimitin order to ensure that no superfluous processing is executed and toprevent noise. SmallLimit may be set to, for instance, 0.0001 in theembodiment. SmallLimit is used in the processing executed in step S190as will be described later. After executing step S20, the operationproceeds to step S30.

In step S30, the control device 104 determines subject positioncoordinates within the target image. In the embodiment, the controldevice 104 prompts the user to indicate a specific subject position inthe target image and sets the coordinates of the position entered by theuser via the operation member 101 as the subject position coordinates.

Following step S30, the operation proceeds to step S40, in which thecontrol device 104 calculates average values for the Y component, the Cbcomponent and the Cr component over a 3×3 (pixel) area centered on thesubject position coordinates having been set in step S30. In thefollowing description, the Y component average value, the Cb componentaverage value and the Cr component average value will be respectivelynotated as aveY, aveCb and aveCr. Once the processing in step S40 isexecuted, the operation proceeds to step S50.

In step S50, the control device 104 makes a decision as to whether ornot the target image is a first frame constituting the live view image,i.e., whether or not the target image is the image in the initial frame.If an affirmative decision is made in step S50, the operation proceedsto step S60. In step S60, the control device 104 crops image portionseach ranging over a predetermined range, e.g., a 180×135 (pixel) rangecentered on the subject position coordinates having been set in stepS30, from the 360×240 (pixel) brightness image (Y image) and chrominanceimages (Cb image and Cr image) having been created in step S10. Thesubsequent processing executed for the three cropped images obtainedthrough this crop processing can be expedited.

Following step S60, the operation proceeds to step S70, in which thecontrol device 104 individually subtracts the Y component average value,the Cb component average value and the Cr component average value havingbeen calculated in step S40, each from the pixel values indicated in thecorresponding cropped image among the three cropped images having beencreated in step S60, and creates three differential images based uponthe absolute values of the differences thus calculated. Differentialimage components DiffY, DiffCb and DiffCr of the differential imageseach corresponding to one of the cropped images are expressed as in (7)through (9) below. The function abs(k) in expressions (7) through (9)represents the absolute value RI of the difference k. As an alternative,differential images may be created by taking the square k² of thedifference k, instead of the absolute value |k| of the difference k.DiffY=abs(Y−aveY)  (7)DiffCb=abs(Cb−aveCb)  (8)DiffCr=abs(Cr−aveCr)  (9)

Once the processing in step S70 is executed, the operation proceeds tostep S80, in which the control device 104 calculates a distanceweighting coefficient Dist, which corresponds to the distance from asubject area assuming a predetermined areal size centered on the subjectposition coordinates set in step S30, e.g., a 30×30 (pixel) rectangulararea, to each pixel position, in each of the three images, i.e., thebrightness image and the two chrominance images. An example of distanceweighting coefficient Dist calculation will be described later. Thecontrol device 104 then creates distance-weighted differential imagesDistY, DistCb and DistCr by individually multiplying the pixel valuesindicated in the differential images expressed with the Y component, theCb component and the Cr component created in step S70 by the distanceweighting coefficient Dist, as expressed in (10) through (12) below.After executing step S80, the operation proceeds to step S140 which willbe explained later. Through the processing executed in step S80, inorder to allow image values in binary images to be obtained throughbinarization, as described later, to readily assume the value of 0,background noise can be eliminated.DistY=DiffY×Dist  (10)DistCb=DiffCb×Dist  (11)DistCr=DiffCr×Dist  (12)

In reference to FIG. 3, an example of distance weighting coefficientDist calculation is described. In the example presented in FIG. 3, anarea 3 has been designated by the control device 104 as the 30×30(pixel) rectangular subject area centered on the subject positioncoordinates. A variable Xsize indicating the size of the rectanglemeasured along an x direction, and a variable Ysize indicating the sizeof the rectangle measured along a y direction, are defined as in (13)below. A variable γ used when calculating the distance weightingcoefficient Dist is defined as in (14) below.Xsize=30,Ysize=30  (13)γ=10×((Xsize/2)²+(Ysize/2)²)  (14)

In the example presented in FIG. 3, the distance weighting coefficientDist is calculated as expressed in (15) below for a pixel present abovethe area 3, e.g., a pixel present in an area 1. The distance weightingcoefficient Dist is calculated as expressed in (16) below for a pixelpresent at a lateral position matching that of the area 3 and present tothe left of the area 3, e.g., a pixel present in an area 2. The distanceweighting coefficient Dist is calculated as expressed in (17) below fora pixel present in the area 3. The distance weighting coefficient Distis calculated as expressed in (18) below for a pixel present at alateral position matching that of the area 3 and present to the right ofthe area 3, e.g., a pixel present in an area 4. The distance weightingcoefficient Dist is calculated as expressed in (19) below for a pixelpresent below the area 3, e.g., a pixel present in an area 5.Dist=1+((y−y coordinate of rectangle upper end)²+(x−x coordinate ofsubject position)²)/γ  (15)Dist=1+((y−y coordinate of subject position)²+(x−x coordinate ofrectangle left end)²)/γ  (16)Dist=1  (17)Dist=1+((y−y coordinate of subject position)²+(x−x coordinate ofrectangle right end)²)/γ  (18)Dist=1+((y−y coordinate of rectangle lower end)²+(x−x coordinate ofsubject position)²)/γ  (19)

If a negative decision is made in step S50, the operation proceeds tostep S90. In step S90, the control device 104 executes processingsimilar to that executed in step S10 for the image newly input from thecontrol device 104 so as to create a brightness image (Y image) and twochrominance images (Cb image and Cr image), the data size of which isreduced to 360×240 (pixel). The operation then proceeds to step S100.

In step S100, in the preceding frame, the control device 104 crops imageportions each ranging over a predetermined range, e.g., a 180×135(pixel) range, centered on the coordinates of the gravitational centerof a mask with the largest evaluation value, having been saved in stepS230 as will be explained later, respectively from the 360×240 (pixel)brightness image and chrominance images having been created in step S90.The subsequent processing executed for the three cropped images obtainedthrough this crop processing is thus expedited. The term “mask” used inthe description of the embodiment refers to a white pixel cluster in abinary image. The mask achieving the largest evaluation value will bedescribed in detail later.

Once the processing in step S100 is executed, the operation proceeds tostep S110, in which the control device 104 creates differential imagesby calculating the differential image components DiffY, DiffCb andDiffCr of the differential images as expressed in (7) through (9) inmuch the same way as it creates the differential images in step S70 asdescribed earlier, each in correspondence to one of the three croppedimages having been created in step S100. The operation then proceeds tostep S120.

In step S120, the control device 104 calculates a mask area of the maskin the preceding frame based upon the coordinates of the four corners ofthe enclosing rectangular subject area enclosing the mask having beensaved in step S230 as will be explained later or obtains the mask areaof the mask in the preceding frame having been saved in step S230 aswill be explained later, and makes a decision as to whether or not themask area of the mask and the entire image plane area of the entirephotographic image plane achieve a relationship expressed in (20) below.mask area/entire image plane area>0.001  (20)

If an affirmative decision is made in step S120, the operation proceedsto step S130, in which the control device 104 determines the enclosingrectangle position in each differential image based upon the coordinatesof the four corners of the enclosing rectangular subject area in thepreceding frame having been saved in step S230, as will be explainedlater, and calculates the distance weighting coefficient Dist incorrespondence to the distance between the enclosing rectangle and eachpixel position. The control device 104 then creates distance-weighteddifferential images DistY, DistCb and DistCr by individually multiplyingthe pixel values indicated in the differential images expressed with theY component, the Cb component and the Cr component having been createdin step S110, by the distance weighting coefficient Dist, as expressedin (10) through (12). Following step S130, the operation proceeds tostep S140.

The distance weighting coefficient Dist can be calculated incorrespondence to the distance from the enclosing rectangle by adoptinga calculation method similar to that described in reference to step 80.For instance, in conjunction with the enclosing rectangle assuming Xsizealong the x direction and Ysize along the y direction, the variable γused when calculating the distance weighting coefficient Dist may bedefined as in (14). For the area 3 in FIG. 4 designated as the enclosingrectangular subject area, the distance weighting coefficient Dist iscalculated as expressed in (15) for a pixel present above the area 3,e.g., a pixel present in an area 1. The distance weighting coefficientDist is calculated as expressed in (16) for a pixel present at a lateralposition matching that of the area 3 and present to the left of the area3, e.g., a pixel present in the area 2. The distance weightingcoefficient Dist is calculated as expressed in (17) for a pixel presentin the area 3. The distance weighting coefficient Dist is calculated asexpressed in (18) for a pixel present at a lateral position matchingthat of the area 3 and present to the right of the area 3, e.g., a pixelpresent in the area 4. The distance weighting coefficient Dist iscalculated as expressed in (19) for a pixel present below the area 3,e.g., a pixel present in an area 5.

If a negative decision is made in step S120, the operation proceeds tostep S140. By executing the distance-based weighting processing in stepS130 only upon making an affirmative decision in step S120 as describedabove, the risk of losing the subject can be lowered since nodistance-based weighting processing is executed for a small object.

In step S140, the control device 104 calculates standard deviationsmanifested by the pixel values in the cropped images having been createdin step S60 or step S100, i.e., a standard deviation sigY manifested inthe Y component cropped image, a standard deviation sigCb manifested inthe Cb-component cropped image and a standard deviation sigCr manifestedin the Cr-component cropped image. In the embodiment, the standarddeviations are calculated based upon the average values aveY, aveCb andaveCr having been calculated over the 3×3 (pixel) range in step S40,instead of the average values for the cropped images. Once theprocessing in step S140 is executed, the operation proceeds to stepS150.

In step S150, the control device 104 binarizes the components DistY,DistCb and DistCr expressing the distance-weighted differential imageshaving been created in step S80 or step S130, as explained below, andthus creates binary images expressed with the Y component, the Cbcomponent and the Cr component. More specifically, the control device104 creates a binary image by binarizing each DistY component value inthe Y component distance-weighted differential image, as expressed in(21) below.if DistY<α×sigY then DistY=1 else DistY=0  (21)

The control device 104 creates a binary image by binarizing each DistCbcomponent value in the Cb component distance-weighted differentialimage, as expressed in (22) below.if DistCb<α×sigCb then DistCb=1 else DistCb=0  (22)

However, if 118<aveCb<138 and sigCb<abs(aveCb−128)+3, the control device104 creates a binary image by binarizing each DistCb component value asexpressed in (23) below.if DistCb<α×sigCb×[{abs(aveCb−128)+3}/sigCb]^(1/2)×[10/{abs(aveCb−128)+0.1}]^(1/2) then DistCb=1 else DistCb=0  (23)

The control device 104 creates a binary image by binarizing each DistCrcomponent value in the Cr component distance-weighted differentialimage, as expressed in (24) below.if DistCr<α×sigCr then DistCr=1 else DistCr=0  (24)

However, if 118<aveCr<138 and sigCr<abs(aveCr−128)+3, the control device104 creates a binary image by binarizing each DistCr component value asexpressed in (25) below.if DistCr<α×sigCr×[{abs(aveCr−128)+3}/sigCr]^(1/2)×[10/{abs(aveCr−128)+0.1}]^(1/2) then DistCr=1 else DistCr=0  (25)

α in expressions (21) through (25) above may assume a value of, forinstance, 0.6. When creating a binary image corresponding to the Cbcomponent and a binary image corresponding to the Cr component, anachromatic subject will not be successfully extracted through standardbinarization if the average value aveCb of the Cb component values isclose to 128 (118<aveCb<138) and the standard deviation sigCb of the Cbcomponent is small (sigCb<abs (aveCb−128)+3) and if the average valueaveCr of the Cr component values is close to 128 (118<aveCr<138) and thestandard deviation sigCr of the Cr component is small (sigCr<abs(aveCr−128)+3). Accordingly, in place of the standard binarization,alternative binarization is executed as expressed in (23) by using aweighting coefficient corresponding to the Cb component average valueaveCb and the standard deviation sigCb and as expressed in (25) by usinga weighting coefficient corresponding to the Cr component average valueaveCr and the standard deviation sigCr.

Once the processing in step S150 is executed, the operation proceeds tostep S160, in which the control device 104 combines binarized Y, Cb andCr images 5 b through 5 d, having been created based upon a target image5 a, through an AND operation as shown in FIG. 5, and thus creates anAND image (synthesized image) 5 e. Based upon the AND image 5 e thuscreated, the subject color can be identified. Following step S160, theoperation proceeds to step S170.

In step S170, the control device 104 executes eight-direction labelingprocessing on the AND image 5 e having been created in step S160. Inthis step, four-direction labeling processing instead of theeight-direction labeling processing may be executed. Following stepS170, the processing proceeds to step S180, in which the control device104 extracts masks constituted with white pixel clusters from thesynthetic image having undergone the labeling processing in step S170and calculates the mask areas of the extracted masks. Once theprocessing in step S180 ends, the operation proceeds to step S190.

In step S190, the control device 104 eliminates any mask that is notlikely to represent a main subject based upon the map areas having beencalculated in step S180. More specifically, the control device 104compares values, each obtained by dividing a given mask area by the ANDimage plane area for the AND image, with a preset lower limitSmallLimit, and retains any mask with the corresponding quotient greaterthan SmallLimit but disqualifies any other mask as a subsequentprocessing target. For the first frame, the value set for SmallLimit instep S20 is used, but the value calculated for SmallLimit in step S250for the preceding frame is used for the second and subsequent frames.Through these measures, faster processing speed is assured bydisqualifying any mask not likely to represent a main subject, e.g., amask that is either too small or too large, as a subsequent processingtarget.SmallLimit<mask area/entire image plane area  (26)

Once the processing in step S190 is executed, the operation proceeds tostep S200 in which the control device 104 sets enclosing rectangularsubject areas that enclose the masks having been labeled in step S170and eliminates any mask not likely to represent a main subject basedupon the aspect ratios of the enclosing rectangles. More specifically,the control device 104 retains any mask satisfying the conditionexpressed in (27) below but disqualifies any other mask as a subsequentprocessing target. Through these measures, faster processing speed isassured by disqualifying any narrow mask not likely to represent a mainsubject as a subsequent processing target.0.2≦longitudinal measurement of enclosing rectangle/lateral measurementof enclosing rectangle≦5  (27)

Following step S200, the operation proceeds to step S210 in which thecontrol device 104 calculates a moment of inertia IM22 for each mask asexpressed in (28) below. Once the processing in step S210 is executed,the operation proceeds to step S220.IM22=ΣΣ{(x−x _(g))²+(y−y _(g))²}  (28)

(x, y) in expression (28) indicates the coordinates of each pixel in agiven mask. (x_(g), y_(g)) indicates the coordinates of the subjectposition having been specified in step S30 if the target image is afirst frame image, whereas it indicates the coordinates of thegravitational center of the mask with the largest evaluation valuehaving been saved through the processing executed in step S230, as willbe described later, for the preceding frame if the target image is asecond or subsequent frame image.

In step S220, the control device 104 calculates an evaluation value foreach mask as expressed in (29) below based upon the corresponding maskarea having been calculated in step S180 and the moment of inertia IM22having been calculated in step S210. β in expression (29) may take avalue of, for instance, 1.5.evaluation value=(mask area)^(β)/IM22  (29)

Once the processing in step S220 is executed, the operation proceeds tostep S230, in which the control device 104 designates a subject byselecting the mask with the largest evaluation value among theevaluation values having been calculated in step S220 as a mask mostlikely to represent the main subject. The control device 104 then savesthe coordinates of the gravitational center of the mask with the largestevaluation value thus selected and the coordinates of the four cornersof the enclosing rectangle enclosing the particular mask having been setin step S200. It also saves the mask area of the mask with the largestevaluation value having been designated. Following step S230, theoperation proceeds to step S240.

In step S240, the control device 104 outputs to the monitor 106 an imageplane with the rectangle enclosing the mask included in the image havingbeen read in step S10 or step S90 based upon the coordinates of thegravitational center of the mask and the coordinates of the four cornersof the enclosing rectangle, which have been saved in step S230. As aresult, the enclosing rectangle indicating the subject position can bedisplayed over the live view image.

Following step S240, the operation proceeds to step S250 in which thecontrol device 104 calculates the value for SmallLimit, to be used instep S190 for the next frame, based upon the areal size of the maskachieving the largest evaluation value among the evaluation valuescalculated in step S220 and saves the SmallLimit value thus calculated.SmallLimit may be calculated as expressed in (30) below.SmallLimit=(mask area/entire image plane area)×0.1  (30)

However, if the relationship expressed as; SmallLimit<InitSmallLimit×0.1exists between the SmallLimit value and InitSmallLimit representingSmallLimit=0.0001 having been set in step S20, SmallLimit should be setequal to InitSmallLimit.

Following step S250, the operation proceeds to step S260, in which thecontrol device 104 tracks the subject. However, tracking processingcannot be executed if there is only one frame and accordingly, theprocessing in step S260 is only executed for the image in the secondframe or a subsequent frame. In step S270, the control device 104 makesa decision as to whether or not the image data input from the imagesensor 103 has ended, i.e., whether or not the current frame is the lastframe. If a negative decision is made in step S270, the operationreturns to step S90 to repeatedly execute the processing. If, on theother hand, an affirmative decision is made in step S270, the processingends. By repeatedly executing the processing up to step S270 as shown inFIGS. 2A and 2B, the subject can be designated in each of the pluralityof frames input in time series and then tracked from one frame to thenext through the plurality of frames in step S260.

The following advantages are achieved with the camera 100 in theembodiment described above.

(1) Based upon the target image, the control device 104 generates abrightness image expressed with the brightness component and chrominanceimages expressed with the chrominance components, determines thecoordinates of a specific subject position in each image among thebrightness image and the chrominance images thus generated, andcalculates an average value for each component among the Y component,the Cb component and the Cr component over a 3×3 (pixels) range centeredon the coordinates of the subject position. The control device 104creates differential images each by subtracting the correspondingaverage value from the pixel values in a cropped image extracted fromone of the images having been generated, i.e., the brightness image andthe chrominance images, binarizes the differential images thus createdand then generates a synthetic image by combining the binary images. Thecontrol device 104 then executes eight-direction or four-directionlabeling processing on the synthetic image (AND image). The controldevice 104 extracts masks constituted with white pixel clusters from thesynthetic image having undergone the processing and designates a subjectin the target image based upon evaluation values calculated for one ofthe masks. As a result, the subject position can be designated by usingsubject color information (chrominance information) and brightnessinformation in any of various photographic scenes.

(2) The control device 104 multiplies each pixel value indicated in agiven differential image by a distance weighting coefficient Distdetermined in correspondence to the distance from a rectangular area(subject area) assuming a predetermined size and centered on the subjectposition coordinates so as to weight the pixel value and generatesweighted differential images each in correspondence to one of the imageshaving been generated, i.e., the brightness image and the chrominanceimages. Since this allows the pixel value indicated at each pixel to beweighted in correspondence to the distance between the subject positionand the position of the particular pixel, the subject position can bedetermined with better accuracy.

(3) The control device 104 prompts the user to indicate the subjectposition in the target image and sets the coordinates of the positionentered by the user via the operation member 101 as the subject positioncoordinates. Through these measures, the subject position coordinatescan be determined with a high level of accuracy.

(4) The control device 104 designates the subject in each of theplurality of frames input in time series. As a result, the subject canbe tracked from one frame to the next through the plurality of frames.

—Variations—

The camera achieved in the embodiment as described above allows for thefollowing variations.

(1) The control device 104 in the embodiment described above crops imageportions over a predetermined range, e.g., a 180×135 (pixel) rangeindividually from the brightness image and the chrominance images instep S60 or step S100 in FIG. 2. As an alternative, the processing inthe subsequent steps may be executed for the brightness image and thechrominance images without cropping image portions from these images.

(2) In the embodiment described above, the control device 104 createsdistance-weighted differential images DistY, DistCb and DistCr each bymultiplying the pixel values indicated in one of the differential imagesexpressed with the Y component, the Cb component and the Cr component bythe distance weighting coefficient Dist in step S80 or step S130 in FIG.2. However, it is not strictly necessary to execute the processing instep S80 or step S130 and the processing in step S140 and subsequentsteps may be executed for the unweighted differential images.

(3) In the embodiment described above, the control device 104 promptsthe user to indicate the subject position in the target image and setsthe coordinates of the position entered by the user via the operationmember 101 as the subject position coordinates. However, the subjectposition coordinates may be set through another method. For instance,the control device 104 may set the coordinates of the AF point for whichfocus adjustment has been executed through autofocus processing, as thesubject position coordinates.

(4) The control device 104 in the embodiment described above calculatesthe average value in correspondence to each component among the Ycomponent, the Cb component and the Cr component over the 3×3 (pixel)range centered on the subject position coordinates indicated by theuser, as a representative value, and generates differential images eachby subtracting the average value, i.e., the representative value, fromthe pixel values indicated in the corresponding cropped image. As analternative, the control device 104 may generate each differential imageby calculating a most frequent value or a median, instead of the averagevalue, as a representative value and subtracting the representativevalue from the pixel values indicated in the cropped image.

(5) The subject designating device and the subject tracking apparatusaccording to the present invention are adopted in a camera in theembodiment described above. However, the present invention is notlimited to this example and it may be adopted in a subject trackingapparatus that reads video data and tracks a subject in the video imagefrom one frame to the next through a plurality of frames, such as apersonal computer or a portable terminal.

As long as the features characterizing the present invention are notcompromised, the present invention is not limited to any of the specificstructural particulars described in reference to the embodiment. Inaddition, the embodiment described above may be adopted in combinationwith a plurality of variations.

The above described embodiment is an example and various modificationscan be made without departing from the scope of the invention.

What is claimed is:
 1. A subject designating device, comprising: a firstimage generation unit that generates a brightness image expressed basedupon a brightness component of a target image and chrominance imagesexpressed based upon chrominance components of the target image; arepresentative value calculation unit that calculates a representativevalue for each image of the brightness image and the chrominance imageshaving been generated by the first image generation unit, based uponpixel values indicated at pixels present within a first subject areacontaining a subject in the each image of the brightness image and thechrominance images; a second image generation unit that creates adifferential image each in correspondence to the each image of thebrightness image and the chrominance images having been generated by thefirst image generation unit, by subtracting the representative valuecalculated by the representative value calculation unit from pixelvalues indicated at pixels present within a second subject areacontaining the subject in the each image of the brightness image and thechrominance images; a binarizing unit that creates a binary image bybinarizing the differential image having been generated by the secondimage generation unit; a synthesizing unit that creates a syntheticimage by combining the binary image created by the binarizing unit incorrespondence to the brightness image and the binary images created bythe binarizing unit in correspondence to the chrominance images; a maskextraction unit that extracts a mask constituted with a white pixelcluster from the synthetic image generated by the synthesizing unit; anevaluation value calculation unit that calculates an evaluation valueindicating a likelihood of the mask, having been extracted by the maskextraction unit, representing the subject; and a subject designatingunit that designates the subject in the target image based upon theevaluation value having been calculated by the evaluation valuecalculation unit.
 2. A subject designating device according to claim 1,further comprising: a third image generation unit that weights eachpixel value indicated at each pixel present in the differential imagegenerated by the second image generation unit by multiplying the eachpixel value by a coefficient determined in correspondence to a distancefrom a third subject area containing the subject in the differentialimage to the each pixel present in the differential image so as togenerate a weighted differential image in correspondence to the eachimage of the brightness image and the chrominance images, wherein: thebinarizing unit binarizes the weighted differential image having beengenerated by the third image generation unit.
 3. A subject designatingdevice according to claim 1, wherein: the representative value is anyone of an average value, a most frequent value and a median of the pixelvalues indicated at the pixels present within the second subject area.4. A subject designating device according to claim 2, wherein: therepresentative value is any one of an average value, a most frequentvalue and a median of the pixel values indicated at the pixels presentwithin the second subject area.
 5. A subject designating deviceaccording to claim 1, wherein: the second subject area is any one of anarea containing a coordinate point on the target image indicated by auser and an area containing an AF point for focus adjustment processing.6. A subject designating device according to claim 2, wherein: thesecond subject area is any one of an area containing a coordinate pointon the target image indicated by a user and an area containing an AFpoint for focus adjustment processing.
 7. A subject designating deviceaccording to claim 3, wherein: the second subject area is any one of anarea containing a coordinate point on the target image indicated by auser and an area containing an AF point for focus adjustment processing.8. A subject designating device according to claim 4, wherein: thesecond subject area is any one of an area containing a coordinate pointon the target image indicated by a user and an area containing an AFpoint for focus adjustment processing.
 9. A subject tracking apparatus,comprising: a subject designating device according to claim 1; and atracking unit that tracks the subject designated by the subjectdesignating unit from one frame to the next through a plurality offrames.
 10. A subject tracking apparatus, comprising: a subjectdesignating device according to claim 2; and a tracking unit that tracksthe subject designated by the subject designating unit from one frame tothe next through a plurality of frames.
 11. A subject trackingapparatus, comprising: a subject designating device according to claim3; and a tracking unit that tracks the subject designated by the subjectdesignating unit from one frame to the next through a plurality offrames.
 12. A subject tracking apparatus, comprising: a subjectdesignating device according to claim 4; and a tracking unit that tracksthe subject designated by the subject designating unit from one frame tothe next through a plurality of frames.
 13. A subject trackingapparatus, comprising: a subject designating device according to claim5; and a tracking unit that tracks the subject designated by the subjectdesignating unit from one frame to the next through a plurality offrames.
 14. A subject tracking apparatus, comprising: a subjectdesignating device according to claim 6; and a tracking unit that tracksthe subject designated by the subject designating unit from one frame tothe next through a plurality of frames.
 15. A subject trackingapparatus, comprising: a subject designating device according to claim7; and a tracking unit that tracks the subject designated by the subjectdesignating unit from one frame to the next through a plurality offrames.
 16. A subject tracking apparatus, comprising: a subjectdesignating device according to claim 8; and a tracking unit that tracksthe subject designated by the subject designating unit from one frame tothe next through a plurality of frames.